An Online Algorithm for Learning Selectivity to Mixture Means

We develop a biologically-plausible learning rule called Triplet BCM that provably converges to the class means of general mixture models. This rule generalizes the classical BCM neural rule, and provides a novel interpretation of classical BCM as performing a kind of tensor decomposition. It achieves a substantial generalization over classical BCM by incorporating triplets of samples from the mixtures, which provides a novel information processing interpretation to spike-timing-dependent plasticity. We provide complete proofs of convergence of this learning rule, and an extended discussion of the connection between BCM and tensor learning. Spectral tensor methods are emerging themes in machine learning, but they remain global rather than “on-line.” While incremental (on-line) learning can be useful in many practical applications, it is essential for biological learning. ∗now at Google Inc. 1 ar X iv :1 41 0. 85 80 v1 [ qbi o. N C ] 3 0 O ct 2 01 4 We introduce a triplet learning rule for mixture distributions based on a tensor formulation of the BCM biological learning rule. It is implemented in a feed forward fashion, removing the need for backpropagation of error signals. Our main result is that a modified version of the classical BienenstockCooper-Munro [3] synaptic update rule, a neuron can perform a tensor decomposition of the input data. By incorporating the interactions between input triplets (commonly referred to as a multi-view assumption), our learning rule can provably learn the mixture means under an extremely broad class of mixture distributions and noise models. This improves on the classical BCM learning rule, which will not converge properly in the presence of noise. We also provide new theoretical interpretations of the classical BCM rule, specifically we show the classical BCM neuron objective function is closely related to some objective functions in the tensor decomposition literature, when the input data consists of discrete input vectors. We also prove convergence for our modified rule when the data is drawn from a general mixture model. The multiview requirement has an intriguing implication for neuroscience. Since spikes arrive in waves, and spike trains matter for learning [7], our model suggests that the waves of spikes arriving during adjacent epochs in time provide multiple samples of a given stimulus. This provides a powerful information processing interpretation to biological learning. To realize it fully, we note that while classical BCM can be implemented via spike timing dependent plasticity [12][8][4][13]. However, most of these approaches require much stronger distributional assumptions on the input data, or learn a much simpler decomposition of the data than our algorithm. Other, Bayesian methods [11], require the computation of a posterior distribution with implausible normalization requirements. Our learning rule successfully avoids these issues, and has provable guarantees of convergence to the true mixture means. This article forms an extended technical presentation of some proofs introduced at NIPS 2014[10], which has more discussion on the implications for bio-